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Abstract 

A recursive approach for shrinking coefficients of an atomic decomposition 
is proposed. The corresponding algorithm evolves so as to provide at each 
iteration a) the orthogonal projection of a signal onto a reduced subspace and 
b) the index of the coefficient to be disregarded in order to construct a coarser 
approximation minimizing the norm of the residual error. 

EDICS Category: 1-TFSR. 

1 Introduction 

Adopting the terminology early introduced in [1] we call atomic decomposition of a 
signal to the linear expansion: 

N 

f = ^2c n a n , (1) 

71=1 

where the atoms a n are elements of a non-orthogonal sequence in the space of the 
signal /, which is assumed to be a Hilbert space 7i. Within the general Match- 
ing Pursuit (MP) framework [1-5] the atoms are chosen, by different criteria, from 
a in general redundant set which is called a dictionary. The problem of selecting 
atoms in order to construct the signal representation with the minimum possible 
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number of coefficients is a very complex problem. In particular, the problem of con- 
structing the optimal approximation with iV-atoms selected so as to minimize the 
Euclidean distance between the signal and the approximation is a NP-hard problem 
[3]. Thus, in this line only suboptimal solutions are actually feasible. In a previ- 
ous publication a suboptimal iterative pursuit strategy, which is only optimal at 
each iteration step, has been introduced with the name of Optimized Orthogonal 
Matching Pursuit (OOMP) approach [5]. Such a technique is based on an iterative 
method for computing dual atoms (3^ ] n = 1, ■ ■ ■ ,N giving rise to a representa- 
tion of the orthogonal projector operator onto the subspace generated by the set 
of atoms a n ; n = 1,...,N. These atoms, which happen to be biorthogonal to 
atoms a n ; n — 1, . . . , N [6], allow to compute the coefficients in (JTJ) by comput- 
ing inner products, i.e. = ((3^ , f) (the superscript N indicates that the dual 
atoms, and therefore the coefficients, depend of the number N of dictionary atoms 
being considered). The OOMP approach tackles the problem of selecting the new 
atom otN+i to improve the approximation. Moreover, the coefficients of the atomic 
decomposition are recursively modified in order to yield an optimal approximation 
in the enlarged subspace. Here we consider the reverse situation: We assume that 
an atomic decomposition is given and we wish to eliminate some coefficients. For 
such an end we propose a technique, that we term Backward Optimized Orthogonal 
Matching Pursuit (BOOMP) which consists of the following elements: a) a recur- 
sive approach to modify the coefficients of the atomic decomposition when one of 
the coefficients is to be disregarded and b) the criterion to select such a coefficient. 
Although the technique can be applied to reduce coefficients of any atomic decom- 
position, regardless of how such a decomposition is obtained, in this letter we focus 
on its implementation as a posterior step of OOMP. The reason for taking this route 
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is that the implementation of BOOMP is really straightforward on the outputs of 
OOMP. In addition, we believe that this way of introducing the approach makes 
more clear its purpose and also its implementation steps. 

We would like to stress that even the construction of suitable suboptimal signal 
approximations by using dictionaries is a complex theoretical and practical chal- 
lenge [7,8]. This communication aims at enhancing the fact that, since suboptimal 
forward and backward approximations are in general not reversible, application of 
the proposed backward pursuit approach, after a forward pursuit selection of atoms, 
may result in a gain with respect to sparseness of the representation. This is clearly 
illustrated here by a simple example. 

2 Adaptive MP strategies 

The MP approach is a technique to compute adaptive signal representations by 
iterative selection of atoms [1]. In its original form this technique does not yield 
at each iteration the linear expansion of the selected atoms that approximate the 
signal at best in a minimum distance sense. A later refinement, which does provide 
such an approximation, has been termed Orthogonal Matching Pursuit (OMP) [2, 
3,9]. However, since OMP selects the atoms according to the MP prescription, 
the selection criterion is not optimal in the sense of minimising the residual of the 
new approximation. The requirement of such minimization has led to the recently 
introduced OOMP approach [5]. This technique is implemented by means of an 
adaptive biorthogonalization method which, within the workings of the selection 
process, generates the set of biorthogonal atoms yielding orthogonal projections 
[5,6]. Such biorthogonal atoms are used to compute the coefficients of the atomic 
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decomposition and are obtained through the following recursive equations: 



where the set i\) j ; j = 1, . . . , k + 1 is obtained inductively, from ipi = cti, by or- 
thogonalization of atoms otj ; j = 1, . . . , k + 1. These atoms are selected from the 
dictionary by minimizing, at each iteration step, the norm of the residual error in 
approximating the signal [5]. 

Let us stress once more that, since all the above mentioned pursuit strategies 
evolve by fixing the atoms selected in the previous steps, there is plenty of room for 
possible improvement with regard to compression of the representation. 

Improving compression after the OOMP procedure implies having to eliminate 
some coefficients of the atomic resulting decomposition. For the coarser approxima- 
tion to be optimal in a minimum distance sense, the remaining coefficients must be 
recalculated [10-12]. This feature of non-orthogonal expansions is a major difference 
with orthogonal ones and has motivated an adaptive approach to modify biorthogo- 
nal atoms in order for then to yield orthogonal projections when the corresponding 
subspace is reduced [12]. 

Let us suppose that OOMP has selected iV atoms to represent a given signal 
up to some predetermined precision. Let us denote Vn to the subspace spanned by 
such atoms i.e, Vn = span{aii, . . . , a^} and let Vn-i be the subspace which is left by 
removing one atom, say the j-th one, i.e. Vjv-i = span{ai, . . . , aj-i, a,-+i, . . . , ajv}- 
Since the biorthogonal atoms (3^ ; n = 1, . . . , N are available (as output of the 
OOMP procedure), to construct the orthogonal projector of / onto V N -i we just 
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(3) 
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need to modify the atoms /3„ ; n = 1, . . . ,j — l,j + 1, . . . ,N as follows [12] 



QNIQN qN\ 

£"" 1 = ^ ||ffl|2 ' n = l,...,j-l,j + l,...,N. (4) 
In writing the above equation we have re-defined the superscript N — 1. Now this 



upper index indicates that the biorthogonal atoms are modified in order to account 
for the deleting of any one atom (not necessarily the last element of the spanning set). 
In the next section we discussed how these adaptive backward equations generate 
the proposed BOOMP approach. 

3 Backward Optimized Orthogonal Matching Pur- 
suit 

After the selection of N atoms the OOMP approach provides a representation of a 
signal / as given by [5] 

N 

fv N = Pv N f = J2 C n a ^ ( 5 ) 
n=l 

Pv N f indicates the orthogonal projection of the signal / onto Vn and coefficients 
c% are obtained as = {(3^,f). Theorem 1 below proves that, if we decide to 
eliminate the coefficient from the above expansion, in order to obtain the optimal 
approximation of / in the reduced subspace Vjv-i, the remaining coefficients ; n = 
1, . . . , j — 1, j + 1, . . . , N should be modified as follows: 



c^^-^-^cf; n 1 • 1 V. (6) 



For the sake of organizing the corresponding proof let us first prove the following 
lemma: 

Lemma 1. Let signal fy be given by 



N 



fo_i = £<£" la »> (7) 



n=l 



with coefficients c^ 1 as in (QJ) ; and let f be a signal in 7i. The difference f — f v _ 
is orthogonal to every function in Vn-i- 

Proof. Using © and (|7J) we have: 

N N N i r>N oN\ 

£ r r N-l £ ST^ N , \Pn >Pj / N 

i ~ f%-i = i -l^ c n a n = f -}_^c n a n + }^ Cj a n 

n=l n=l n=l ""i " 

N N ioN oN\ 

— J C n a n + Cj Otj + ^ 1 1 R N\ |2 S' °" C j a J " 



n=l 



- Il/3f 



Since (3f G Vjv, it follows from © that J^ n=1 pyjp ct n = = pyjja - Hence 

(El turns out to be 



/-/^^/-E^n + Cf^. (9) 

Since by hypothesis 5^ n=1 a n = Pv N f, the difference / — J2n=i c n a n is orthogonal 
to every function in V^. Furthermore, since (cx n , (3^) = 5 n j, by taking the inner 
product both sides of with every function a n ; n = 1, . . . , j — 1, j + 1, . . . , N 
we obtain (a n , f — fy N _) = 0, which proves that / — fv„ is orthogonal to every 
function in Vn-x- □ 

Lemma 2. The coefficients c^ -1 of the linear expansion 

N 

fv N ^ = Y, C n~ la n (10) 

71=1 

minimizing the distance in Vn-i to a given signal f 6 Ti. are obtainable, from 
and Pf? j n = 1, . . . , N , as prescribed in (OJ). 

Proof. Let g be an arbitrary signal in Vjv_i and let us write || / — g\\ 2 as follows: 

n/-^ir = n/-/v JV „ 1 +/v' JV _ 1 -^ii 2 - (ii) 
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From Lemma ^ we know that / — f v is orthogonal to every function in V/v-i and 
since fy N1 — g is in Vjv_i we have 

ll/-^l| 2 = ll/-/y iV „ 1 l| 2 + ll/y iV _ 1 -^l| 2 , (12) 
from where we conclude that ||/ — g\\ 2 is minimized if g = fy ■ □ 

LemmaEltells us how to proceed to disregard coefficients of a non-orthogonal lin- 
ear expansion. Assuming that the coefficient to be disregarded has been selected, 
in order to optimize the approximation in a minimum distance sense, the remain- 
ing coefficients should be modified as indicated in (jSJ. The next theorem gives an 
answer to the question as to how to select the coefficient to be neglected. 

Theorem 1. Let Rj be the residual resulting by disregarding a coefficient for 
passing from approximation fy N to fy _ i-c, fy N = f v _ +Rj. In order to minimize 
the norm of the residual Rj such coefficient is to be chosen as the one yielding a 
minimum value of the quantity 



\ r N\2 
\ 3 I 
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(13) 



Proof. Since Rj = fy N - U = J2n=i c n a n - E«=i c n b y usin g © we have: 



N N N i oN oN\ 

JV Kr-'n i r J j i 

j an WW 



n=l n=l n=l 



cfa 3 



(B N B N ) 

+ c NS Ta - n - j ' -c N a- (14) 

^ C 3 n \\RN\\2 3 3 \ ' 



N 



As already discussed, ^2 n =i a n(Pn > Pf) = Pf- Then, from the last equation it 

c N /3 N 

follows that Rj = pjvjp . Consequently, in order to minimize ||-Rj|| 2 the coefficient 
cf to be neglected is the one minimizing (|13p. □ 



Theorem ^ leads to a recursive algorithm for shrinking coefficients. We call such 
algorithm BOOMP, because, at each iteration, it selects the atom to be deleted 



according to a selection criterion which is equivalent to the one proposed by OOMP 
[5] for forward approximations. Moreover, BOOMP is a natural complement of 
OOMP because its implementation on the output of OOMP is extremely simple. 
The few necessary steps are describe below. 

BOOMP algorithm 

Let us assume that atoms a n ; n = 1, . . . , N have been selected by the OOMP 
approach in order to approximate a signal /. Hence, the biorthogonal set f3^ and 
the corresponding coefficients ; n — 1, . . . , N are also known. The BOOMP 
approach for reducing coefficients evolves as follows: 

• Select the index j of the coefficient to be disregarded as the one yielding a 

\c N \ 2 

minimum value of the quantity pjkjp as j ranges from 1 to N. 

• Modify the corresponding biorthogonal atoms and coefficients as prescribed in 
(J1J) and (P) respectively. 

• Set N = N — 1 and repeat the above steps until the coarsest acceptable 
approximation is reached. 

Example 

We illustrate now by a simple example the main remark of this communication: 
namely that BOOMP can improve the compression performance of the forward 
OOMP approach. We construct a dictionary of Mexican hat wavelets given by the 
functions 

2 i t 2 

a m ,n(t) = 2^a(t2 m - 0.2n) with a(t) = -^tt~*(1 _ t 2 )e~- . (15) 

v3 

By considering scales m = 0,1,2,3, and 4 to cover the [0,4] interval we have a 
dictionary of 665 atoms. The signal / to be represented is a chirp generated by the 
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MATLAB instructions: 
t=0:0.01:4; f =chirp (t , , 1 , 2) ; 

In order to produce a high quality representation of this chirp the OOMP approach 
selects A = 60 atoms (the norm of the residual error is 0.0544). By applying the 
proposed BOOMP to the OOMP approximation we reduce the number of coeffi- 
cients up to 34 and the approximation is the one depicts in the top left graph of 
Figure 1. The norm of the residual error with respect to the true signal is in this case 
1.18. However, if rather than applying the BOOMP approach we stop the OOMP 
approach at iteration 34, so as to have the same number of coefficients as in the 
previous case, the approximation is the one shown in the top right graph of Figure 1 
and the norm of the residual error is 1.77. It is clear from the graphs that, in addi- 
tion to yielding the smallest residual error with the same number of coefficients, the 
approximation obtained by the BOOMP approach is overall visually superior to the 
OOMP one with the same number of coefficients. The bottom left and right graph 
depict, respectively, the absolute value of the difference between the chirp signal and 
the corresponding approximations. 

4 Conclusions 

A recursive approach for shrinking coefficients of an atomic decomposition has been 
proposed. The approach is based on an adaptive technique which allows to modify 
biorthogonal functions in order to yield orthogonal projectors onto a reduced sub- 
space. A criterion for disregarding coefficients has been discussed. Such criterion 
leads to an iterative procedure that we have termed BOOMP, because it evolves 
so as to fulfil identical requirements to those of the OOMP method. Accordingly, 
BOOMP provides at each iteration a) the coefficient of the atomic decomposition 
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to be deleted in order to construct a coarser approximation minimizing the norm 
of the residual error b) the coefficients of such decomposition rendering optimal 
approximation in the same sense. The approach is a good complement to OOMP, 
very simple to implement and definitely worth trying in any case. However it is 
appropriate to stress that situations for which the forward approach renders better 
results that the combination with the backward one should certainly exist. This 
is a consequence of lack of global optimality in both directions. Finally we would 
like to remark that if, rather than (|13|) . one decided to apply another criterion for 
disregarding coefficients (see [13, 14] for some alternative ones) in order to leave 
an approximation minimizing the distance to the signal the remaining coefficients 
should be modified as prescribed in ©• 
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Figure 1: The top left graph represents the approximation obtained by applying 
the BOOMP approach for reducing the 60 coefficients of the high quality OOMP 
approximation up to 34. The top right graph corresponds to the OOMP approxima- 
tion resulting by stopping the approach after the selection of 34 atoms. The bottom 
left graph depicts the absolute value of the difference between the chirp signal and 
the BOOMP approximation of the left top graph. The right bottom graph has the 
same description as the left one, but with respect to the approximation of the right 
top graph. 
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